Skip to content

nsenter: fix FUSE deadlock from synchronous child waits#121

Open
benjamincburns wants to merge 2 commits into
nestybox:masterfrom
benjamincburns:fix/fuse-deadlock
Open

nsenter: fix FUSE deadlock from synchronous child waits#121
benjamincburns wants to merge 2 commits into
nestybox:masterfrom
benjamincburns:fix/fuse-deadlock

Conversation

@benjamincburns

@benjamincburns benjamincburns commented May 26, 2026

Copy link
Copy Markdown

Fixes nestybox/sysbox#998. Builds on #119.

SendRequest() held the reaper RLock while synchronously waiting for nsenter child processes. If a child's exit was delayed by fuse_flush on inherited FUSE file descriptors, the RLock was held indefinitely, blocking both the reaper and new FUSE handlers.

This PR includes the TryLock fix from #119 and adds:

  • Move pipe read before PARENT wait (PARENT writes pid before exiting)
  • Replace synchronous Process.Wait() calls with background goroutines
  • Add reapProcessAsync for the INIT process to trigger a reaper sweep on exit
  • Regression tests for the changes from nsenter: use TryLock in reaper to prevent cascading FUSE deadlock #119 and the additional commit included in this PR.

Just for transparency, I am an experienced software engineer with some systems experience, but I'm not a go dev. I used AI tooling (Claude Code and Claude Opus 4.6) to find the underlying deadlock and author this fix. I'm mentioning that here in case this project has any policies against AI contributions. I don't want to subvert any such policy.

I have tested this fix on my own system. See notes in nestybox/sysbox#998 (comment).

@okhowang @benjamincburns
nsenter: use TryLock in reaper to prevent cascading FUSE deadlock
8808abf 
Replace mu.Lock() with mu.TryLock() in the zombie reaper goroutine. Go's
sync.RWMutex implements writer-preference: a pending Lock() blocks all
subsequent RLock() callers. When the reaper's Lock() is pending while an
nsenter child holds RLock, all new FUSE Lookup handlers are blocked from
acquiring RLock to start their own nsenter processes. If the nsenter child's
operation triggers a FUSE request back to sysbox-fs, the FUSE handler cannot
proceed, creating a deadlock.

TryLock() does not register a pending writer, so FUSE handlers can still
acquire RLock. If TryLock fails, the reaper sleep a second to retry.

Fixes: nestybox/sysbox#1002
Signed-off-by: okhowang(王沛文) <okhowang@tencent.com>
@benjamincburns

benjamincburns commented May 26, 2026

Copy link
Copy Markdown
Author

cc @okhowang for visibility

@benjamincburns
nsenter: async child waits to prevent FUSE deadlock
8ddc7c6 
SendRequest() held the reaper RLock while synchronously waiting for
nsenter child processes via Process.Wait(). If a child's exit was
delayed by the kernel's fuse_flush on inherited FUSE file descriptors,
the RLock was held indefinitely, preventing both the reaper and new
FUSE handlers from making progress.

Move the pipe read before the PARENT wait (the PARENT writes the pid
before exiting) and replace synchronous Process.Wait() calls with
background goroutines so the caller releases the RLock promptly.

Fixes: nestybox/sysbox#998
Signed-off-by: Ben Burns <803016+benjamincburns@users.noreply.github.com>
@benjamincburns benjamincburns mentioned this pull request Jun 1, 2026
Open
@danieleds

danieleds commented Jun 3, 2026

Copy link
Copy Markdown

I've tested this PR and it fully solves the problem for me as far as I can tell!

@0xACCE55

0xACCE55 commented Jun 7, 2026

Copy link
Copy Markdown

Independent confirmation from a separate Sysbox CE 0.7.0 / kernel-6.8 deployment (Coder-style workspaces, inner dockerd + compose stack). Same hang: load into the dozens, idle CPU, procs in D, workspace stuck at inner-dockerd start.

What we measured, three ways:

  • Kernel stacks: the nsenter agent stages (runc:[0:PARENT]/[1:CHILD]) wedge in fuse_flush on exit; sysbox-fs handler threads sit in waitid (the Process.Wait()s).
  • Live fuse_flush disasm (/proc/kcore, 6.8): agents block at fuse_flush+0xff = the return of call down_write at +0xfa, i.e. inode_lock(inode) on the FUSE inode's exclusive i_rwsem.
  • sysbox-fs core (goroutine walk): ~10 handlers blocked in SendRequest's Process.Wait(); nothing parked on the reaper RWMutex — so in our case the live edge was the inode lock, not reaper-lock starvation.

Cycle: sysbox-fs sets OpenDirectIO and not FOPEN_PARALLEL_DIRECT_WRITES, so fuse_direct_write_iter holds the inode i_rwsem exclusive across the whole request. A workspace's write to an emulated /proc/sys file holds it and waits on sysbox-fs → the handler blocks in Process.Wait() for the agent to exit → the agent's exit-time fuse_flush blocks on inode_lock() for that same inode.

This PR fixes it regardless of variant: the real problem is the handler blocking on the child's exit at all. Making the waits async lets the handler answer the request, the client drops inode_lock, and the agent's flush can then complete — severing both the inode-lock cycle and the RWMutex one. Might be worth noting the inode-lock manifestation in the description, but the code covers both.

Verified: #119+#121 on the 0.7.0 commit, go test ./nsenter/ green, deployed — the stop/start that reliably reproduced the wedge for us now boots clean (intermittent race, so not proof, but no recurrence with a detector armed). Full writeup + disasm + core-walker available if useful.

(For transparency, per the note above: also done with Claude Code, arrived at independently.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Hangs when running simultaneous docker containers inside a sysbox container

4 participants

@benjamincburns @danieleds @0xACCE55 @okhowang